Multi-degrees-of-freedom hand held controller

ABSTRACT

A controller including a first control member and a second control member that extends from a portion of the first control member. A controller processor is operable to produce a rotational movement output signal in response to movement of the first control member, and a translational movement output signal in response to movement of the second control member relative to the first control member. The first control member may be gripped and moved using a single hand, and the second control member may be moved using the thumb of the single hand.

This application claims the benefit of U.S. provisional patentapplication No. 62/413,685 filed Oct. 27, 2016, the entirety of which isincorporated herein by reference for all purposes.

FIELD OF THE INVENTION

The present disclosure relates generally to control systems and moreparticularly to a controller that provides a user with the ability tosend command signals for up to six independent degrees of freedom,substantially limiting cross-coupling, using a controller that isoperable with a single hand.

BACKGROUND OF THE INVENTION

Conventionally, multiple discrete controllers are utilized to allow auser to control a control target having more than three degrees offreedom. Furthermore, multiple discrete controllers have been requiredfor any conventional control system that controls a control targethaving six degrees of freedom. For example, a set of independentcontrollers or input devices (e.g., joysticks, control columns, cyclicsticks, foot pedals, and/or other independent controllers as may beknown by one or more of ordinary skill in the art) may be provided toreceive a variety of different rotational parameters (e.g., pitch, yaw,and roll) from a user for a control target (e.g., an aircraft,submersible vehicles, spacecraft, a control target in a virtualenvironment, and/or a variety of other control targets as may be knownby one or more of ordinary skill in the art). Similarly, a set ofindependent controllers may be provided to control other navigationalparameters such as translation (e.g., x-, y-, and z-axis movement) in athree-dimensional (3D) space, velocity, acceleration, and/or a varietyof other command parameters.

U.S. patent application Ser. Nos. 13/797,184 and 15/071,624,respectively filed on Mar. 12, 2013, and Mar. 16, 2016, which are bothincorporated herein by reference in their entireties, describe severalembodiments of a control system that allows a user to control a controltarget in up to six degrees of freedom (6-DOF) simultaneously andindependently, using a single controller. In one embodiment, a unifiedhand controller may include a first control member for receivingrotational inputs (e.g., pitch, yaw, and roll) and a second controlmember that extends from the first control member and that is forreceiving translational inputs (e.g., movement along x, y, and z axes).The first control member and the second control member on the unifiedhand controller may be repositioned by a user using a single hand tocontrol the control target in 6-DOF. Vital to precise and intuitivecontrol is knowledge of when a command is being issued in a particulardegree of freedom and when it is not. Each independent controllerprovides tactile feedback to the user, such that s/he knows when inputsare being made in up to six degrees of freedom motion, simultaneouslyand independently.

SUMMARY

Previously known drone, virtual reality, augmented reality, computer andgaming input devices are not intuitive, require substantial initial andproficiency training, and are operated with two hands. They are alsotypically not mobile.

Various aspects of the single-handled controllers described below canallow a computer augmented or virtual reality gamer or otherusers-in-motion (hikers, skiers, security/SAR personnel, war-fighters,and others, for example) to control assets in physical and/or virtualthree-dimensional space, by enabling up to 6-DoF motion in all axessimultaneously while also limiting cross-coupling (unintended motions).

According to one aspect of the disclosure, a hand controller includesfirst, second, and third control members. The first control member ismovable with three degrees of freedom and provides in response a firstset of three independent control inputs. Movement of the first membermay be sensed, and control inputs generated, by, for example, aninertial motion unit, potentiometers gimbals or a combination thereof.The first control member is configured to be gripped in a user's singlehand, by the user placing it in the palm of the hand and wrapping atleast several of their fingers at least partially around the body of thefirst member to hold it. The second control member is disposed on a topend of the first member, near where the thumb of a hand might rest whenthe first member is gripped, and is movable with three independentdegrees of freedom independently of the movement of the first controlmember. In response to its independent degrees of freedom, the secondcontrol member provides a second set of three independent controlinputs. The control inputs of the second set are independent of thecontrol inputs of the first set, and the second control member isconfigured to be manipulated by the thumb of the user's hand that isgripping of the first control member.

Extended operation of a controller with a second member with a thumb forindependent control inputs might lead to thumb fatigue. The thirdcontrol member is positioned to be manipulated by one or more digitsother than the index finger of the user's single hand and is coupledwith the second member to move in opposition to movement of the secondcontrol member in one of the degrees of freedom of the second controlmember. The third control member, such as a paddle, is mounted on thefirst member in a position for the third, fourth and fifth digits on theuser's hand (or a sub-set of these) to squeeze or rotate the thirdmember. The third member is coupled to the second member to push it up,along a Z-axis, when the third member is moved to thereby hold the thumbup. The thumb is less likely to be fatigued. Pushing down the secondcontrol member may, if desired, also push outwardly from the controllerthe third control member, allowing the thumb and accessory digits to bein a dynamic balance.

A controller with these features can be used to allow the controller todecouple translation from attitude adjustments in the controlrequirements of computer aided design, drone flight, various types ofcomputer games, virtual and augmented reality and other virtual andphysical tasks where precise movement through space is required.

In a separate aspect of the disclosure, a hand controller having atleast first and second control members (and, optionally, a third controlmember), which is configured for gripping by a user's single hand, maybe coupled with a wrist or forearm brace that serves as a reference forrotational axes, particularly yaw. Yaw is difficult to measure with aninertial measurement unit (IMU) within a hand-held controller. Forexample, although an IMU in the hand controller might be able to senseand measure with sufficient precision and sensitivity pitch and roll(rotation about the X and Y axes) of the first member, it has been foundthat outputs of an IMU for rotation about the Z-axis corresponding toyaw of the first control member can be noisy. A linkage between thefirst control member and a user's wrist or forearm and a potentiometer,optical encoder, or other types of sensor for measuring rotation can beused to sense yaw.

According to one aspect of the several representative embodimentsdescribed below, a single-handed controller that mounts on the wrist andthat registers displacement from a neutral position defined relative tothe wrist, allowing flight, gaming or augmented reality motion controlin up to six degrees of freedom of motion (6-DoF) with precision.Passive mechanical, vibration haptic or active mechanical feedback mayinform the user of their displacement from zero in each of these 6-DoF.With such a single-handed control, movement through the air like afighter pilot with intuitive (non-deliberate cognitive) inputs ispossible.

In accordance with another aspect of the disclosure, a forearm bracecoupled with a controller can used in combination with an index fingerloop to open or close a grasp on an object in a virtual world.

Another aspect of different ones of the representative embodiments ofhand controllers described below, involves a two-handed controller thatprovides a consistent, known reference frame stabilized by thenon-dominant hand even while moving, e.g., walking, skiing, running,driving. The hand controller can be plugged into the surface of a base,allowing the non-flying hand to stabilize the base as it is being flown.Moving a point of reference (POR) through physical or virtual space byway of a hand controller raises the problem of requiring insight intodisplacement in every degree of freedom being controlled so that thelocation of the “zero input” is known for each degree of freedom. Forexample, for drones, the zero input positions for x, y, and z axes andyaw need to be always known. Other flight regimes, such as virtual andaugmented reality, computer gaming and surgical robotics may require asmany as six independent degrees of freedom simultaneously (movementalong x, y, and z axes, and pitch, yaw, and roll). Moreover, for droneflight and virtual and augmented reality systems in particular, theability to be mobile while maintaining precise control of the point ofreference is desirable.

In one of these representative embodiments, a first control member inthe form of a joystick mounted to a base allows for pitch, yaw and rollinputs where it connects to the base, with centering mechanisms togenerate forces to inform the user of zero. A second control member ontop of the joystick, in a position that can operated with a thumb (orpossibly another digit), being displaced or moved along x and y axes togenerate control outputs for x-axis and y-axis movement and beingdisplaceable up or down to receive input for z-axis movement

Various aspects, advantages, features and embodiments are included inthe following description of exemplary examples thereof, whichdescription should be taken in conjunction with the accompanyingdrawings. All patents, patent applications, articles, otherpublications, documents and things referenced herein are herebyincorporated herein by this reference in their entirety for allpurposes. To the extent of any inconsistency or conflict in thedefinition or use of terms between any of the incorporated publications,documents or things and the present application, those of the presentapplication shall prevail.

BRIEF DESCRIPTION OF THE DRAWINGS

For promoting an understanding of the principles of the invention thatis claimed below, reference will now be made to the embodiments, orexamples, illustrated in the appended drawings. It will be understoodthat, by describing specific embodiments and examples, no limitation ofthe scope of the invention, beyond the express terms set out in theclaims, is intended. Alterations and further modifications to thedescribed embodiments and examples are possible while making use of theclaimed subject matter, and therefore are contemplated as being withinthe scope of the invention as claimed.

FIG. 1 is a schematic view of an embodiment of a control system.

FIG. 2 is a flowchart illustrating an embodiment of a method forcontrolling a control target.

FIG. 3 is a side view illustrating an embodiment of a user using thecontroller depicted in FIG. 2A-FIG. 2G with a single hand.

FIG. 4A is a side view illustrating an embodiment of a physical orvirtual vehicle control target executing movements according to themethod of FIG. 2.

FIG. 4B is a top view illustrating an embodiment of the physical orvirtual vehicle control target of FIG. 4A executing movements accordingto the method of FIG. 4A.

FIG. 4C is a front view illustrating an embodiment of the physical orvirtual vehicle control target of FIG. 4A executing movements accordingto the method of FIG. 2.

FIG. 4D is a perspective view illustrating an embodiment of a toolcontrol target executing movements according to the method of FIG. 2.

FIG. 5 is a flowchart illustrating an embodiment of a method forcontrolling a control target.

FIG. 6 is a flowchart illustrating an embodiment of a method forconfiguring a controller.

FIG. 7 is a side view of a first, representative embodiment of asingle-hand controller.

FIG. 8A is a perspective view of a second, representative embodiment ofa single-hand controller that is partially assembled, with a pivotingplatform for a second control member in a first position.

FIG. 8B is a perspective view of the second, representative embodimentof a single-hand controller that is partially assembled, with thepivoting platform for the second control member in a second position.

FIG. 8C is a perspective view of the second, representative is aperspective view of the second, representative embodiment of asingle-hand controller in a different state of assembly than shown inFIGS. 8A and 8B, with one-half of a housing forming a first controlmember removed.

FIG. 9 illustrates a perspective view of a third, representativeembodiment of a controller having a secondary control member in the formof a thumb loop.

FIG. 10 illustrates a perspective view of a fourth, representativeembodiment of a controller having a gantry-type secondary controlmember.

FIG. 11 illustrates a perspective view of a fifth, representativeembodiment of a controller having a trackball-type secondary controlmember.

FIG. 12 is a perspective view of a mobile, two-handed control systemhaving a controller mounted to a base.

FIG. 13 is a perspective view of a controller mounted to a base havinginput buttons.

FIG. 14 is a perspective view of a single-handed controller mounted to awired base.

FIG. 15 is a perspective illustration of another, representative exampleand embodiment single-handed controller that is amounted to a bracketconnected with a user's forearm.

FIG. 16 is a perspective view of yet another representative example andembodiment of a hand controller connected with to a forearm attachmentworn by a user.

FIG. 17 is a perspective view of a representative example of a handlecontroller coupled with a cuff mounted on a user's forearm.

FIG. 18 is a side view of the representative example of a handlecontroller coupled with a cuff mounted on a user's forearm shown in FIG.17.

FIG. 19A is a top view of a representative example of a control systemhaving a double-gimbal link between a forearm attachment and a handcontroller.

FIG. 19B is a side view of the control system of FIG. 19A.

FIG. 19C is a perspective view of the control system of FIG. 19A.

FIG. 19D is a perspective view of a second, representative example of acontrol system having a double-gimbal link between a forearm attachmentand a hand controller.

FIG. 20A is a side view of another, representative example of a controlsystem of a control system having a double-gimbal link between a forearmattachment and a hand controller.

FIG. 20B is a different side view of the control system of FIG. 20A.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

In the drawings and description that follows, the drawings are notnecessarily to scale. Certain features of the invention may be shownexaggerated in scale or in somewhat schematic form and some details ofconventional elements may not be shown in the interest of clarity andconciseness. The present disclosure is susceptible to embodiments ofdifferent forms. Specific embodiments are described in detail and areshown in the drawings, with the understanding that the presentdisclosure is to be considered an exemplification of the principles ofthe invention, and is not intended to limit the invention to thatillustrated and described herein. It is to be fully recognized that thedifferent teachings of the embodiments discussed below may be employedseparately or in any suitable combination to produce desired results.The various characteristics mentioned above, as well as other featuresand characteristics described in more detail below, will be readilyapparent to those skilled in the art upon reading the followingdescription of illustrative embodiments of the invention, and byreferring to the drawings that accompany the specification.

The present disclosure describes several embodiments of a control systemthat allows a user to control a control target in up to six degrees offreedom (6-DOF) using a single controller. In one embodiment, a unifiedhand controller may include a first control member for receiving a firstset of one, two or three inputs from a user and a second control memberthat extends from the first control member that can receive a second setof one, two or three additional inputs from the user. These controllermaps inputs to preselected outputs that are used to control a targetcontrol system. The first control member and the second control memberon the unified hand controller may be repositioned by a user using asingle hand to control the control target in up to six degrees offreedom.

More specifically, in some of the embodiments of a control systemdescribed below, a user is able to control a control target in six DOF(6-DOF) using a single controller. In one embodiment, a unified handcontroller may include a first control member for receiving rotationalinputs (e.g., pitch, yaw, and roll) and a second control member thatextends from the first control member and that is for receivingtranslational inputs (e.g., movement along x, y, and z axes). Asdescribed in further detail below, the first control member and thesecond control member on the unified hand controller may be repositionedby a user using a single hand to control the control target in 6-DOF.

The embodiments described below are examples of an improved single-handcontroller with one or more additional features as compared to prior arthand controllers. These additional features and enhancements include:improved Z-axis spring forces and self-centering/zeroing capability fora second member that is controlled by a user's thumb when gripping afirst member of a controller; a larger gantry on top of first member formoving the second member in along X and Y axes; a replaceable orresizable thumb loop for the second control member; a forearm or wriststabilization for ambulatory use (potentiometers or optical encoders fortranslations along X, Y and Z axes, such as for use in droneapplications and for integrating with virtual/augmented reality); amouse-based implementation for improved CAD object manipulation; andcombinations of any two or more of the preceding features.

The hand controller with any one or more of these features, and theirvariations, can be used in applications such as flight simulation,computer aided design (CAD), drone flight, fixed wing and rotary wingflight, computer gaming, virtual and augmented reality navigation,terrestrial and marine robotic control, and many others, some of whichare described below.

Referring initially to FIG. 1, a control system 100 for controlling acontrol target in 6-DOF. The control system 100 includes a controller102 that is coupled to a signal conversion system 104 that is furthercoupled to a control target 106. In an embodiment, the control target106 may include end effectors (e.g., the end of a robotic forceps, arobotic arm end effector with snares), camera field-of-views (e.g.,including a camera center field-of-view and zoom), vehicle velocityvectors, etc. While the controller 102 and the signal conversion system104 are illustrated separately, one of ordinary skill in the art willrecognize that some or all of the controller 102 and the signalconversion system 104 may be combined without departing from the scopeof the present disclosure.

The controller 102 includes a first control member 102 a and a secondcontrol member 102 b that is located on the first control member 102 a.In this description, controller 102 is intended to be representative ofthe all of the controllers described herein, unless otherwise indicated.A controller processor 102 c is coupled to each of the first controlmember 102 a and the second control member 102 b. In an embodiment, thecontroller processor 102 c may be a central processing unit, aprogrammable logic controller, and/or a variety of other processors asmay be known by one or more of ordinary skill in the art. The controllerprocessor 102 c is also coupled to each of a rotational module 102 d, atranslation module 102 e, and a transmitter 102 f. While not illustratedor described in any further detail, other connections and coupling mayexist between the first control member 102 a, the second control member102 b, the controller processor 102 c, the rotation module 102 d, thetranslation module 102 e, and the transmitter 102 f while remainingwithin the scope of the present disclosure. Furthermore, components ofthe controller may be combined or substituted with other components asmay be known by one or more of ordinary skill in the art while remainingwith the scope of the present disclosure.

The signal conversion system 104 in the control system 100 includes atransceiver 104 a that may couple to the transmitter 102 f in thecontroller 102 through a wired connection, a wireless connection, and/ora variety of other connections as may be known by one or more ofordinary skill in the art. A conversion processor 104 b is coupled tothe transceiver 104 a, a control module 104 c, and configurationparameters 104 d that may be included on a memory, a storage device,and/or other computer-readable mediums as may be known by one or more ofordinary skill in the art. In an embodiment, the conversion processor104 b may be a central processing unit, a programmable logic controller,and/or a variety of other processors known to those of ordinary skill inthe art. While not illustrated or described in any further detail, otherconnections and coupling may exist between the transceiver 104 a, theconversion processor 104 b, the control module 104 c, and theconfiguration parameters 104 d while remaining within the scope of thepresent disclosure. Furthermore, components of the signal conversionsystem 104 may be combined or substituted with other components as maybe known by one or more of ordinary skill in the art while remainingwith the scope of the present disclosure. The control module 104 c maybe coupled to the control target 106 through a wired connection, awireless connection, and/or a variety of other connections as may beknown by one or more of ordinary skill in the art.

In an embodiment, the controller 102 is configured to receive input froma user through the first control member 102 a and/or the second controlmember 102 b and transmit a signal based on the input. For example, thecontroller 102 may be provided as a “joystick” for navigating in avirtual environment (e.g., in a video game, on a real-world simulator,as part of a remote control virtual/real-world control system, and/or ina variety of other virtual environments as may be known by one or moreof ordinary skill in the art.) In another example, the controller 102may be provided as a control stick for controlling a vehicle (e.g., anaircraft, a submersible, a spacecraft, and/or a variety of othervehicles as may be known by one or more of ordinary skill in the art).In another example, the controller 102 may be provided as a controlstick for controlling a robot or other non-vehicle device (e.g., asurgical device, an assembly device, and/or variety of other non-vehicledevices known to one of ordinary skill in the art).

In the embodiment discussed in further detail below, the controller 102includes a control stick as the first control member 102 a that isconfigured to be repositioned by the user. The repositioning of thecontrol stick first control member 102 a allows the user to providerotational inputs using the first control member 102 a that includepitch inputs, yaw inputs, and roll inputs, and causes the controllerprocessor 102 c to output rotational movement output signals includingpitch movement output signals, a yaw movement output signals, and rollmovement output signals. In particular, tilting the control stick firstcontrol member 102 a forward and backward may provide the pitch inputthat produces the pitch movement output signal, rotating the controlstick first control member 102 a left and right about its longitudinalaxis may provide the yaw input that produces the yaw movement outputsignal, and tilting the control stick first control member 102 a side toside may provide the roll input that produces the roll movement outputsignal. As discussed below, the movement output signals that result fromthe repositioning of the first control member 102 a may be reconfiguredfrom that discussed above such that similar movements of the firstcontrol member 102 a to those discussed above result in different inputsand movement output signals (e.g., tilting the control stick firstcontrol member 102 a side to side may be configured to provide the yawinput that produces the yaw movement output signal while rotating thecontrol stick first control member 102 a about its longitudinal axis maybe configured provide the roll input that produces the roll movementoutput signal.)

Rotational inputs using the control stick first control member 102 a maybe detected and/or measured using the rotational module 102 d. Forexample, the rotational module 102 d may include displacement detectorsfor detecting the displacement of the control stick first control member102 a from a starting position as one or more of the pitch inputs, yawinputs, and roll inputs discussed above. Displacement detectors mayinclude photo detectors for detecting light beams, rotary and/or linearpotentiometers, inductively coupled coils, physical actuators,gyroscopes, switches, transducers, and/or a variety of otherdisplacement detectors as may be known by one or more of ordinary skillin the art. In some embodiments, the rotational module 102 d may includeaccelerometers for detecting the displacement of the control stick firstcontrol member 102 a from a starting position in space. For example, theaccelerometers may each measure the proper acceleration of the controlstick first control member 102 a with respect to an inertial frame ofreference.

In other embodiments, inputs using the control stick first controlmember 102 a may be detected and/or measured using breakout switches,transducers, and/or direct switches for each of the three ranges ofmotion (e.g., front to back, side to side, and rotation about alongitudinal axis) of the control stick first control member 102 a. Forexample, breakout switches may be used to detect when the control stickfirst control member 102 a is initially moved (e.g., 2°) from a nullposition for each range of rotation, transducers may provide a signalthat is proportional to the displacement of the control stick firstcontrol member 102 a for each range of motion, and direct switches maydetect when the control stick first control member 102 a is furthermoved (e.g., 12°) from the null position for each range of motion. Thebreakout switches and direct switches may also allow for acceleration ofthe control stick first control member 102 a to be detected. In anembodiment, redundant detectors and/or switches may be provided in thecontroller 102 to ensure that the control system 100 is fault tolerant.

In the embodiment discussed in further detail below, the second controlmember 102 b extends from a top, distal portion of the control stickfirst control member 102 a and is configured to be repositioned by theuser independently from and relative to the control stick first controlmember 102 a. The repositioning of the second control member 102 bdiscussed below allows the user to provide translational inputs usingthe second control member 102 b that include x-axis inputs, y-axisinputs, and z-axis inputs, and causes the control processor 102 c tooutput a translational movement output signals including x-axis movementoutput signals, y-axis movement output signals, and z-axis movementoutput signals. For example, tilting the second control member 102 bforward and backward may provide the x-axis input that produces thex-axis movement output signal, tilting the second control member 102 bside to side may provide the y-axis input that produces the y-axismovement output signal, and moving the second control member 102 b upand down may provide the z-axis input that produces the z-axis movementoutput signal. As discussed below, the signals that result from therepositioning of the second control member 102 b may be reconfiguredfrom that discussed above such that similar movements of the secondcontrol member 102 b to those discussed above result in different inputsand movement output signals (e.g., tilting the second control member 102b forward and backward may be configured to provide the z-axis inputthat produces the z-axis movement output signal while moving the secondcontrol member 102 b up and down may be configured to provide the x-axisinput that produces the x-axis movement output signal.) In anembodiment, the second control member 102 b is configured to berepositioned solely by a thumb of the user while the user is grippingthe control stick first control member 102 a with the hand that includesthat thumb.

Translational inputs using the second control member 102 b may bedetected and/or measured using the translation module 102 e. Forexample, the translation module 102 e may include translationaldetectors for detecting the displacement of the second control member102 b from a starting position as one or more of the x-axis inputs,y-axis inputs, and z-axis inputs discussed above. Translation detectorsmay include physical actuators, translational accelerometers, and/or avariety of other translation detectors as may be known by one or more ofordinary skill in the art (e.g., many of the detectors and switchesdiscussed above for detecting and/or measuring rotational input may berepurposed for detecting and/or measuring translation input.)

In an embodiment, the controller processor 102 c of the controller 102is configured to generate control signals to be transmitted by thetransmitter 102 f. As discussed above, the controller processor 102 cmay be configured to generate a control signal based on one or morerotational inputs detected and/or measured by the rotational module 102d and/or one or more translational inputs detected and/or measured bythe translation module 102 e. Those control signal generated by thecontroller processor 102 c may include parameters defining movementoutput signals for one or more of 6-DOF (i.e., pitch, yaw, roll,movement along an x-axis, movement along a y-axis, movement along az-axis). In several embodiments, a discrete control signal type (e.g.,yaw output signals, pitch output signals, roll output signals, x-axismovement output signals, y-axis movement output signals, and z-axismovement output signals) is produced for each discrete predefinedmovement (e.g., first control member 102 a movement for providing pitchinput, first control member 102 a movement for providing yaw input,first control member 102 a movement for providing roll input, secondcontrol member 102 b movement for providing x-axis input, second controlmember 102 b movement for providing y-axis input, and second controlmember 102 b movement for providing z-axis input) that produces thatdiscrete control signal. Beyond 6-DOF control, discrete features such asON/OFF, trim, and other multi-function commands may be transmitted tothe control target. Conversely, data or feedback may be received on thecontroller 102 (e.g., an indicator such as an LED may be illuminatedgreen to indicate the controller 102 is on.)

In an embodiment, the transmitter 102 f of the controller 102 isconfigured to transmit the control signal through a wired or wirelessconnection. For example, the control signal may be one or more of aradio frequency (“RF”) signal, an infrared (“IR”) signal, a visiblelight signal, and/or a variety of other control signals as may be knownby one or more of ordinary skill in the art. In some embodiments, thetransmitter 102 f may be a BLUETOOTH® transmitter configured to transmitthe control signal as an RF signal according to the BLUETOOTH® protocol(BLUETOOTH® is a registered trademark of the Bluetooth Special InterestGroup, a privately held, not-for-profit trade association headquarteredin Kirkland, Wash., USA).

In an embodiment, the transceiver 104 a of the signal conversion system104 is configured to receive the control signal transmitted by thetransmitter 102 f of the controller 102 through a wired or wirelessconnection, discussed above, and provide the received control signal tothe conversion processor 104 b of the signal conversion system 104.

In an embodiment, the conversion processor 104 b is configured toprocess the control signals received from the controller 102. Forexample, the conversion processor 104 b may be coupled to acomputer-readable medium including instructions that, when executed bythe conversion processor 104 b, cause the conversion processor 104 b toprovide a control program that is configured to convert the controlsignal into movement commands and use the control module 104 c of thesignal conversion system 104 to control the control target 106 accordingto the movement commands. In an embodiment, the conversion processor 104b may convert the control signal into movement commands for a virtualthree-dimensional (“3D”) environment (e.g., a virtual representation ofsurgical patient, a video game, a simulator, and/or a variety of othervirtual 3D environments as may be known by one or more of ordinary skillin the art.). Thus, the control target 106 may exist in a virtual space,and the user may be provided a point of view or a virtual representationof the virtual environment from a point of view inside the controltarget (i.e., the control system 100 may include a display that providesthe user a point of view from the control target in the virtualenvironment). In another example, the control target 106 may be aphysical device such as a robot, an end effector, a surgical tool, alifting system, etc., and/or a variety of steerable mechanical devices,including, without limitation, vehicles such as unmanned orremotely-piloted vehicles (e.g., “drones”); manned, unmanned, orremotely-piloted vehicles and land-craft; manned, unmanned, orremotely-piloted aircraft; manned, unmanned, or remotely-pilotedwatercraft; manned, unmanned, or remotely-piloted submersibles; as wellas manned, unmanned, or remotely-piloted space vehicles, rocketry,satellites, and such like.

In an embodiment, the control module 104 c of the signal conversionsystem 104 is configured to control movement of the control target 106based on the movement commands provided from the control program insignal conversion system 104. In some embodiments, if the control target106 is in a virtual environment, the control module 104 c may include anapplication programming interface (API) for moving a virtualrepresentation or point of view within the virtual environment. API'smay also provide the control module 104 c with feedback from the virtualenvironment such as, for example, collision feedback. In someembodiments, feedback from the control target 106 may allow the controlmodule 104 c to automatically adjust the movement of the control targetto, for example, avoid a collision with a designated region (e.g.,objects in a real or virtual environment, critical regions of a real orvirtual patient, etc.). In other embodiments, if the control target 106is a physical device, the control module 104 c may include one or morecontrollers for controlling the movement of the physical device. Forexample, the signal conversion system 104 may be installed on-board avehicle, and the control module 104 c may include a variety of physicalcontrollers for controlling various propulsion and/or steeringmechanisms of the vehicle.

In an embodiment, the signal conversion system 104 includes operatingparameters 104 d for use by the conversion processor 104 b whengenerating movement commands using the signals from the controller 102.Operating parameters may include, but are not limited to, gains (i.e.,sensitivity), rates of onset (i.e., lag), deadbands (i.e., neutral),limits (i.e., maximum angular displacement), and/or a variety of otheroperating parameters as may be known by one or more of ordinary skill inthe art. In an embodiment, the gains of the first control member 102 aand the second control member 102 b may be independently defined by auser. In this example, the second control member 102 b may haveincreased sensitivity compared to the control stick first control member102 a to compensate, for example, for the second control member 102 bhaving a smaller range of motion that the control stick first controlmember 102 a. Similarly, the rates of onset for the first control member102 a and the second control member 102 b may be defined independentlyto determine the amount of time that should pass (i.e., lag) before arepositioning of the first control member 102 a and the second controlmember 102 b should be converted to actual movement of the controltarget 106. The limits and deadbands of the first control member 102 aand the second control member 102 b may be independently defined as wellby calibrating the neutral and maximal positions of each.

In an embodiment, operating parameters may also define how signals sentfrom the controller 102 in response to the different movements of thefirst control member 102 a and the second control member 102 b aretranslated into movement commands that are sent to the control target.As discussed above, particular movements of the first control member 102a may produce pitch, yaw, and roll rotational movement output signals,while particular movements of the second control member 102 b mayproduce x-axis, y-axis, and z-axis translational movement outputsignals. In an embodiment, the operating parameters may define whichmovement commands are sent to the control target 106 in response tomovements and resulting movement output signals from the first controlmember 102 a and second control member 102 b.

A single hand controller like the ones described shown in FIGS. 7-20B,can provide up to 6 degrees of freedom control. For applications in manytypes of physical and virtual 3-D environments, all 6 degrees of freedommay be required, such as moving a spacecraft or many types of aircraft,or certain computer games and virtual reality and augmented realityenvironments. In many of these cases, the best way to manage them is tomap the x-axis, y-axis, and z-axis translational output signalsgenerated by displacement of the second control member to x-axis, y-axisand z-axis movements in the target application, and use the pitch, rolland yaw rotational output signals generated by displacement of the firstcontrol member to provide rotational control output signals that controlpitch, roll and yaw in the target application.

However, for many other applications like drone flight, when only 4command axes are needed, a user's inputs might be split in differentways, depending whether the hand controller is mounted on a fixed basefor the controller, stabilized by the non-dominant hand, or coupled witha forearm brace. For example, when using a forearm brace to support thehand controller and provide a frame of reference, it might be moredesirable to control the y-axis movement of the drone using the secondmember, but use the first control member to control x-axis movement andyaw. Because the controller's individual input “devices” are easilyprogrammable, you have the ability to choose whatever combination ofinputs and axes you′d like.

In some embodiments, the configuration parameters 104 d may be receivedfrom an external computing device (not shown) operated by the user. Forexample, the external computing device may be preconfigured withsoftware for interfacing with the controller 102 and/or the signalconversion system 104. In other embodiments, the configurationparameters 104 d may be input directly by a user using a display screenincluded with the controller 102 or the signal conversion system 104.For example, the first control member 102 a and/or second control member102 b may be used to navigate a configuration menu for defining theconfiguration parameters 104 d.

Referring now to FIGS. 2 and 3, a method 400 for controlling a controltarget is illustrated using one of as single hand controller. Theillustrated controller in FIG. 3 is representative of single handcontrollers having a first control member gripped by a user's hand,which can be displaced to generate a first set of control outputs and asecond control member that is positioned on the first control member, tobe manipulated by the thumb on the hand gripping the first controlmember, to generate a second set of control outputs. Any of the singlehand controllers described herein may be used with the t methodsdescribed in connection with these figures, unless otherwisespecifically stated. As is the case with the other methods describedherein, various embodiments may not include all of the steps describedbelow, may include additional steps, and may sequence the stepsdifferently. Accordingly, the specific arrangement of steps shown inFIG. 2 should not be construed as limiting the scope of controlling themovement of a control target.

The method 400 begins at block 402 where an input is received from auser. As previously discussed, a user may grasp the first control memberwith a hand, while using a thumb on a second control member. Asillustrated in FIG. 3, a user may grasp the first control member 204with a hand 402 a, while extending a thumb 402 b through the thumbchannel defined by the second control member 208. Furthermore, the usermay position a finger 402 c over the control button 206. One of ordinaryskill in the art will recognize that while a specific embodiment havingthe second control member 208 positioned for thumb actuation and controlbutton 206 for finger actuation are illustrated, other embodiments thatinclude repositioning of the second control member 208 (e.g., foractuation by a finger other than the thumb), repositioning of thecontrol button 206 (e.g., for actuation by a finger other than thefinger illustrated in FIG. 3), additional control buttons, and a varietyof other features will fall within the scope of the present disclosure.

In an embodiment, the input from the user at block 402 of the method 400may include one or more rotational inputs (i.e., a yaw input, a pitchinput, and a roll input) and one or more translational inputs (i.e.,movement along an x-axis, a y-axis, and/or a z-axis) that are providedby the user using, for example, the controllers. The user may repositionthe first control member to provide rotational inputs and reposition thesecond control member to provide translational inputs. The controller is“unified” in that it is capable of being operated by a single hand ofthe user. In other words, the controller allows the user tosimultaneously provide rotational and translational inputs with a singlehand without cross-coupling inputs (i.e., the outputs from the handcontroller are “pure”).

As discussed above, the rotational and translational input may bedetected using various devices such as photo detectors for detectinglight beams, rotary and/or linear potentiometers, inductively coupledcoils, physical actuators, gyroscopes, accelerometers, and a variety ofother devices as may be known by one or more of ordinary skill in theart. A specific example of movements of the first control member and thesecond control member and their results on the control target 106 arediscussed below, but as discussed above, any movements of the firstcontrol member and the second control member may be reprogrammed orrepurposed to the desires of the user (including reprogramming referenceframes by swapping the coordinate systems based on the desires of auser), and thus the discussion below is merely exemplary of oneembodiment of the present disclosure.

As illustrated in FIG. 3, the user may use his/her hand 402 a to movethe first control member 204 back and forth along a line A (e.g., on itscoupling to the base 202 for the controller 200, by tilting the gripportion 204 c that is coupled to the first section 204 b of the firstcontrol member 204 along the line A relative to the bottom portion ofthe first control member 204 for the controller 200), in order toprovide pitch inputs to the controller 200. The grip portion 204 c ofthe first control member 204 includes a top surface 204 d that islocated opposite the grip portion 204 c from the first section of 204 bof the first control member 204. As illustrated in FIG. 3, the user mayuse his/her hand 402 a to rotate the first control member 204 back andforth about its longitudinal axis on an arc B (e.g., on its coupling tothe base 202 for the controller 200, by rotating the grip portion 204 cof the first control member 204 in space for the controller 300), inorder to provide yaw inputs to the controller 200 or 300. As illustratedin FIG. 3, the user may use their hand 402 a to move the first controlmember 204 side to side along a line C (e.g., on its coupling to thebase 202 for the controller 200, by tiling the grip portion 204 c of thefirst control member 204 along the line B relative to the bottom portionof the first control member 204 for the controller 200), in order toprovide roll inputs to the controller 200. Furthermore, additionalinputs may be provided using other features of the controller 200. Forexample, the resilient member may provide a neutral position of thefirst control member 204 such that compressing the resilient memberusing the first control member 204 provides a first input and extendingthe resilient member using the first control member 204 provides asecond input.

The second control member 208 includes a support portion 208 b thatextends from the first control member. The second control member alsoincludes an actuation portion 208 c that is coupled to the supportportion of 208 b of the first control member 204. As illustrated in FIG.3, the user may use the thumb 402 b to move the second control member208 forwards and backwards along a line E (e.g., on its coupling to thefirst control member 204), in order to provide x-axis inputs to thecontroller 200. As illustrated in FIG. 3, the user may use the thumb 402b to move the second control member 208 back and forth along a line D(e.g., on its coupling to the first control member 204), in order toprovide y-axis inputs to the controller 200. As illustrated in FIG. 3,the user may use the thumb 402 b to move the second control member 208up and down along a line F (e.g., on its coupling to the first controlmember 204 including, in some embodiments, with resistance from theresilient member), in order to provide z-axis inputs to the controller200. In an embodiment, the resilient member may provide a neutralposition of the second control member 208 such that compressing theresilient member using the second control member 208 provides a firstz-axis input for z-axis movement of the control target 106 in a firstdirection, and extending the resilient member using the second controlmember 208 provides a second z-axis input for z-axis movement of thecontrol target 106 in a second direction that is opposite the firstdirection.

The method 400 then proceeds to block 404 where a control signal isgenerated based on the user input received in block 402 and thentransmitted. As discussed above, the controller processor 102 c and therotational module 102 d may generate rotational movement output signalsin response to detecting and/or measuring the rotational inputsdiscussed above, and the control processor 102 c and the translationmodule 102 e may generate translational movement output signals inresponse to detecting and/or measuring the translation inputs discussedabove. Furthermore, control signals may include indications of absolutedeflection or displacement of the control members, rate of deflection ordisplacement of the control members, duration of deflection ordisplacement of the control members, variance of the control membersfrom a central deadband, and/or a variety of other control signals knownin the art.) For example, control signals may be generated based on therotational and/or translational input or inputs according to theBLUETOOTH® protocol. Once generated, the control signals may betransmitted as an RF signal by an RF transmitter according to theBLUETOOTH® protocol. Those skilled in the art will appreciate that an RFsignal may be generated and transmitted according to a variety of otherRF protocols such as the ZIGBEE® protocol, the Wireless USB protocol,etc. In other examples, the control signal may be transmitted as an IRsignal, a visible light signal, or as some other signal suitable fortransmitting the control information. (ZIGBEE® is a registered trademarkof the ZigBee Alliance, an association of companies headquartered in SanRamon, Calif., USA).

The method 400 then proceeds to block 406 where a transceiver receives asignal generated and transmitted by the controller. In an embodiment,the transceiver 102 of the signal conversion system 104 receives thecontrol signal generated and transmitted by the controller 102, 200,300. In an embodiment in which the control signal is an RF signal, thetransceiver 104 a includes an RF sensor configured to receive a signalaccording to the appropriate protocol (e.g., BLUETOOTH®, ZIGBEE®,Wireless USB, etc.).

In other embodiments, the control signal may be transmitted over a wiredconnection. In this case, the transmitter 102 f of the controller 102and the transceiver 104 a of the signal conversion system 104 may bephysically connected by a cable such as a universal serial bus (USB)cable, serial cable, parallel cable, proprietary cable, etc.

The method 400 then proceeds to block 408 where control program providedby the conversion processor 104 b of the signal conversion system 104commands movement based on the control signals received in block 406. Inan embodiment, the control program may convert the control signals tomovement commands that may include rotational movement instructionsand/or translational movement instructions based on the rotationalmovement output signals and/or translational movement output signals inthe control signals. Other discrete features such as ON/OFF, camerazoom, share capture, and so on can also be relayed. For example, themovement commands may specify parameters for defining the movement ofthe control target 106 in one or more DOF. Using the example discussedabove, if the user uses their hand 402 a to move the first controlmember 204 back and forth along a line A (illustrated in FIG. 3), theresulting control signal may be used by the control program to generatea movement command including a pitch movement instruction for modifyinga pitch of the control target 106. If the user uses their hand 402 a torotate the first control member 204 back and forth about itslongitudinal axis about an arc B (illustrated in FIG. 3), the resultingcontrol signal may be used by the control program to generate a movementcommand including a yaw movement instruction for modifying a yaw of thecontrol target 106. If the user uses their hand 402 a to move the firstcontrol member 204 side to side along a line C (illustrated in FIG. 3),the resulting control signal may be used by the control program togenerate a movement command including a roll movement instruction formodifying a roll of the control target 106.

Furthermore, if the user uses their thumb 402 b to move the secondcontrol member 208 forward and backwards along a line E (illustrated inFIG. 3), the resulting control signal may be used by the control programto generate a movement command including an x-axis movement instructionfor modifying the position of the control target 106 along an x-axis. Ifthe user uses their thumb 402 b to move the second control member 208back and forth along a line E (illustrated in FIG. 3), the resultingcontrol signal may be used by the control program to generate a movementcommand including a y-axis movement instruction for modifying theposition of the control target 106 along a y-axis. If the user usestheir thumb 402 b to move the second control member 208 side to sidealong a line D (illustrated in FIG. 3), the resulting control signal maybe used by the control program to generate a movement command includinga z-axis movement instruction for modifying the position of the controltarget 106 along a z-axis.

The method 400 then proceeds to block 410 where the movement of thecontrol target 106 is performed based on the movement commands. In anembodiment, a point of view or a virtual representation of the user maybe moved in a virtual environment based on the movement commands atblock 410 of the method 400. In another embodiment, an end effector, apropulsion mechanism, and/or a steering mechanism of a vehicle may beactuated based on the movement commands at block 410 of the method 400.

FIG. 4A, FIG. 4B, and FIG. 4C illustrate a control target 410 a that maybe, for example, the control target 106 discussed above, with referenceto FIG. 1. As discussed above, the control target 410 a may include aphysical vehicle in which the user is located, a remotely operatedvehicle where the user operates the vehicle remotely from the vehicle, avirtual vehicle operated by the user through the provision of apoint-of-view to the user from within the virtual vehicle, and/or avariety of other control targets as may be known by one or more ofordinary skill in the art. Using the example above, if the user usestheir hand 402 a to move the first control member 204 back and forthalong a line A (illustrated in FIG. 3), the movement command resultingfrom the control signal generated will cause the control target 410 a tomodify its pitch about an arc AA, illustrated in FIG. 4B. If the useruses their hand 402 a to rotate the first control member 204 back andforth about its longitudinal axis about an arc B (illustrated in FIG.3), the movement command resulting from the control signal generatedwill cause the control target 410 a to modify its yaw about an arc BB,illustrated in FIG. 4B. If the user uses their hand 402 a to move thefirst control member 204 side to side along a line C (illustrated inFIG. 3), the movement command resulting from the control signalgenerated will cause the control target 410 a to modify its roll aboutan arc CC, illustrated in FIG. 4C.

Furthermore, if the user uses his/her thumb 402 b to move the secondcontrol member 208 forward and backwards along a line E (illustrated inFIG. 3), the movement command resulting from the control signalgenerated will cause the control target 410 a to move along a line EE(i.e., its x-axis), illustrated in FIG. 4B and FIG. 4C. If the user useshis/her thumb 402 b to move the second control member 208 side to sidealong a line D (illustrated in FIG. 3), the movement command resultingfrom the control signal generated will cause the control target 410 a tomove along a line DD (i.e., its y-axis), illustrated in FIG. 4A and FIG.4B. If the user uses his/her thumb 402 b to move the second controlmember 208 back and forth along a line F (illustrated in FIG. 3), themovement command resulting from the control signal generated will causethe control target 410 a to move along a line FF (i.e., its z-axis),illustrated in FIG. 4A and FIG. 4C. In some embodiments, the controlbutton 206 and/or other control buttons on the controller 102, 200, or300 may be used to, for example, actuate other systems in the controltarget 410 a (e.g., weapons systems.)

FIG. 4D illustrates a control target 410 b that may be, for example, thecontrol target 106 discussed above, with reference to FIG. 1. Asdiscussed above, the control target 410 b may include a physical deviceor other tool that executed movements according to signals sent from thecontroller 102, 200, or 300. Using the example above, if the user usestheir hand 402 a to move the first control member 204 back and forthalong a line A (illustrated in FIG. 3), the movement command resultingfrom the control signal generated will cause the control target 410 b torotate a tool member or end effector 410 c about a joint 410 d along anarc AAA, illustrated in FIG. 4D. If the user uses their hand 402 a torotate the first control member 204 back and forth about itslongitudinal axis about an arc B (illustrated in FIG. 3), the movementcommand resulting from the control signal generated will cause thecontrol target 410 b to rotate the tool member or end effector 410 cabout a joint 410 e along an arc BBB, illustrated in FIG. 4D. If theuser uses his/her hand 402 a to move the first control member 204 sideto side along a line C (illustrated in FIG. 3), the movement commandresulting from the control signal generated will cause the controltarget 410 b to rotate the tool member or end effector 410 c about ajoint 410 f along an arc CCC, illustrated in FIG. 4D.

Furthermore, if the user uses his/her thumb 402 b to move the secondcontrol member 208 forwards and backwards along a line E (illustrated inFIG. 3), the movement command resulting from the control signalgenerated will cause the tool member or end effector 410 c to move alonga line EEE (i.e., its x-axis), illustrated in FIG. 4D. If the user useshis/her thumb 402 b to move the second control member 208 back and forthalong a line E (illustrated in FIG. 3), the movement command resultingfrom the control signal generated will cause the control target 410 b tomove along a line EEE (i.e., its y-axis through the joint 410 f),illustrated in FIG. 4D. If the user uses his/her thumb 402 b to move thesecond control member 208 side to side along a line D (illustrated inFIG. 3), the movement command resulting from the control signalgenerated will cause the tool member or end effector 410 c to move alonga line DDD (i.e., its z-axis), illustrated in FIG. 4D. In someembodiments, the control button 206 and/or other control buttons on thecontroller 102, 200, or 300 may be used to, for example, perform actionsusing the tool member 210 c. Furthermore one of ordinary skill in theart will recognize that the tool member or end effector 410 cillustrated in FIG. 4D may be replaced or supplemented with a variety oftool members (e.g., surgical instruments and the like) without departingfrom the scope of the present disclosure. As discussed above, thecontrol target 410 a may include a camera on or adjacent the tool memberor end effector 410 c to provide a field of view to allow navigation toa target.

Referring now to FIG. 5, a method 500 for controlling a control targetis illustrated. As is the case with the other methods described herein,various embodiments may not include all of the steps described below,may include additional steps, and may sequence the steps differently.Accordingly, the specific arrangement of steps shown in FIG. 5 shouldnot be construed as limiting the scope of controlling the movement of acontrol target.

The method 500 may begin at block 502 where rotational input is receivedfrom a user. The user may provide rotational input by repositioning thefirst control member 204 of the controller 200 or 300 similarly asdiscussed above. In some embodiments, the rotational input may bemanually detected by a physical device such as an actuator. In otherembodiments, the rotational input may be electrically detected by asensor such as an accelerometer.

The method 500 may proceed simultaneously with block 504 wheretranslational input is received from the user. The user may providetranslational input by repositioning the second control member 208 ofthe controller 200 or 300 similarly as discussed above. The rotationalinput and the translational input may be provided by the usersimultaneously using a single hand of the user. In some embodiments, thetranslational input may be manually detected by a physical device suchas an actuator.

In an embodiment, the rotational and translational input may be providedby a user viewing the current position of a control target 106 on adisplay screen. For example, the user may be viewing the currentposition of a surgical device presented within a virtual representationof a patient on a display screen. In this example, the rotational inputand translational input may be provided using the current view on thedisplay screen as a frame of reference.

The method 500 then proceeds to block 506 where a control signal isgenerated based on the rotational input and translational input and thentransmitted. In the case of the rotational input being manuallydetected, the control signal may be generated based on the rotationalinput and translational input as detected by a number of actuators,which convert the mechanical force being asserted on the first controlmember 204 and the second control member 208 to an electrical signal tobe interpreted as rotational input and translational input,respectively. In the case of the rotational input being electronicallydetected, the control signal may be generated based on rotational inputas detected by accelerometers and translational input as detected byactuators.

In an embodiment, a control signal may be generated based on therotational input and translational input according to the BLUETOOTH®protocol. Once generated, the control signal may be transmitted as an RFsignal by an RF transmitter according to the BLUETOOTH® protocol. One ofordinary skill in the art will appreciate that an RF signal may begenerated and transmitted according to a variety of other RF protocolssuch as the ZIGBEE® protocol, the Wireless USB protocol, etc. In otherexamples, the control signal may be transmitted as an IR signal, visiblelight signal, or as some other signal suitable for transmitting thecontrol information.

The method 500 then proceeds to block 508, the transceiver 104 a of thesignal conversion system 104 receives the control signal. In the casethat the control signal is an RF signal, the transceiver 104 a includesan RF sensor configured to receive a signal according to the appropriateprotocol (e.g., BLUETOOTH®, ZIGBEE®, Wireless USB, etc.). In otherembodiments, the control signal may be transmitted over a wiredconnection. In this case, the transmitter 102 f and the transceiver 104a are physically connected by a cable such as a universal serial bus(USB) cable, serial cable, parallel cable, proprietary cable, etc.

The method 500 then proceeds to block 510 where the conversion processor104 b commands movement in 6 DOF based on the received control signal.Specifically, the control signal may be converted to movement commandsbased on the rotational and/or translational input in the controlsignal. The movement commands may specify parameters for defining themovement of a point of view or a virtual representation of the user inone or more DOF in a virtual 3D environment. For example, if the secondcontrol member is repositioned upward by the user, the resulting controlsignal may be used to generate a movement command for moving a point ofview of a surgical device up along the z-axis within a 3D representationof a patient's body. In another example, if the first control member istilted to the left and the second control member is repositioneddownward, the resulting control signal may be used to generate movementcommands for rolling a surgical device to the left while moving thesurgical device down along a z-axis in the 3D representation of thepatient's body. Any combination of rotational and translational inputmay be provided to generate movement commands with varying combinationsof parameters in one or more DOF.

The method 500 then proceeds to block 512 where a proportional movementis performed in the virtual and/or real environment based on themovement commands. For example, a point of view of a surgical device ina virtual representation of a patient may be repositioned according tothe movement commands, where the point of view corresponds to a cameraor sensor affixed to a surgical device. In this example, the surgicaldevice may also be repositioned in the patient's body according to themovement of the surgical device in the virtual representation of thepatient's body. The unified controller allows the surgeon to navigatethe surgical device in 6-DOF within the patient's body with a singlehand.

Referring now to FIG. 6, a method 600 for configuring a controller isillustrated. As is the case with the other methods described herein,various embodiments may not include all of the steps described below,may include additional steps, and may sequence the steps differently.Accordingly, the specific arrangement of steps shown in FIG. 6 shouldnot be construed as limiting the scope of controlling the movement of acontrol target.

The method 600 begins at block 602 where the controller 102 is connectedto an external computing device. The controller 102 may be connected viaa physical connection (e.g., USB cable) or any number of wirelessprotocols (e.g., BLUETOOTH® protocol). The external computing device maybe preconfigured with software for interfacing with the controller 102.

The method 600 then proceeds to block 604 where configuration data isreceived by the controller 102 from the external computing device. Theconfiguration data may specify configuration parameters such as gains(i.e., sensitivity), rates of onset (i.e., lag), deadbands (i.e.,neutral), and/or limits (i.e., maximum angular displacement). Theconfiguration data may also assign movement commands for a controltarget to movements of the first control member and second controlmember. The configuration parameters may be specified by the user usingthe software configured to interface with the controller 102.

The method 600 then proceeds to block 606 where the operating parametersof the controller 102 are adjusted based on the configuration data. Theoperating parameters may be stored in memory and then used by thecontroller 102 to remotely control a control target as discussed abovewith respect to FIG. 2 and FIG. 5. In some embodiments, the method 600may include the ability to set “trim”, establish rates of translation(e.g., cm/sec) or reorientation (e.g., deg/sec), or initiate“auto-sequences” to auto-pilot movements (on a display or on thecontroller 102 itself.)

In other embodiments, the controller 102 may be equipped with an inputdevice that allows the user to directly configure the operatingparameters of the controller 102. For example, the controller 102 mayinclude a display screen with configuration menus that are navigableusing the first control member 204 and/or the second control member 208.

A computer readable program product stored on a tangible storage mediamay be used to facilitate any of the preceding embodiments such as, forexample, the control program discussed above. For example, embodimentsof the invention may be stored on a computer readable medium such as anoptical disk e.g., compact disc (CD), digital versatile disc (DVD),etc., a diskette, a tape, a file, a flash memory card, or any othercomputer readable storage device. In this example, the execution of thecomputer readable program product may cause a processor to perform themethods discussed above with respect to FIG. 2, FIG. 5, and FIG. 6.

In the following examples of single hand controllers, various aspectsallow the controller to separate individual translation from attitudeadjustments in the control requirements of computer aided design, droneflight, various types of computer games, virtual and augmented realityand other virtual and physical tasks where precise movement throughspace is required, while simultaneously providing tactile feedback whenaway from the “null command” or zero input position.

For example, extended operation of a controller using the thumb forindependent control inputs can lead to a “hitchhiker's thumb” fatigueissue. By adding a third control member, such as a linked paddle for the3rd, 4th and 5th digits (or some sub-set of these) of the user's hand tosqueeze or rotate while gripping the first control member, the secondcontroller can be held up or pushed up (in +z direction), thus providingrelief. Furthermore, the third control member and the second controlmember can be linked so that pushing down the second control memberpushes out the paddle or third control member. As such, the thumb andaccessory digits are in a dynamic balance, which can be quicklymastered.

In other embodiments, the single hand controller can be used as part ofa control system that has a wrist or forearm brace to serve as areference for the rotational axes, particularly yaw that is difficult tomeasure with an inertial measurement unit (IMU). For example, althoughan IMU within the body of the first control member of the handcontroller may work well for pitch and roll, but yaw can be noisy.Although this may be improved with software modifications, someexemplary embodiments described herein have a linkage to the wristallows for potentiometers or optical encoders to measure all threerotational axes with precision. In some variants of a forearm braceimplementation can use an index finger loop, used to open or close agrasp on an object in a virtual world.

The hand controller examples presented in connection with FIGS. 7-20Band their variations can be used in applications such as those presentedabove in the preceding section, such flight simulation, CAD, droneflight, and so on. Optional additional features, which may be used aloneor, in several case, in combination with one or more of the otherfeatures, include: adjustable z spring forces and self-centering/zeroingcapability; a relatively large x-y gantry on top of joystick for thesecond control member; a replaceable or resizable thumb loop for thesecond control member; forearm or wrist stabilization for ambulatory use(potentiometers or optical encoders for X/Y/Z translations, such as foruse in drone applications and for integrating with virtual/augmentedreality); and a mouse-based implementation for improved CAD objectmanipulation.

Referring now FIGS. 7 to 11, controllers 700, 900, 1000 and 1100illustrate different, representative embodiments of a single-handcontroller having three control members, one of which provides Z-axissecondary control.

The exemplary controllers 700, 900, 1000, 1000, as well as thecontrollers shown and described in FIGS. 12-20B, translational inputsfor indicating movement along the X, Y and Z axes are preferablyreceived from a user's the thumb. The thumb is mapped to the brain ingreatest detail relative to other parts of the hand. These controllersexploit its greater dexterity to provide input along the X, Y, and Zaxes. As the thumb movements are relative to the first control member,which in is these examples are in the form of a joy stick, translationcan be decoupled from attitude control of the target control object.Squeezing a third control member located on the first control memberallows any one or more of the third, fourth or fifth digits on a user'shand to support the user's thumb by applying an upward force or upwardmotion. The force and movement of the third control member istransmitted or applied to the second control member, and thus to thethumb, through an internal coupling.

These embodiments use an inertial measurement units for measuringdisplacements of the first control member. However, in alternative,these controllers can be adapted to use external sensors when thecontroller is mounted to pivot on a base, in which case sensors forsensing roll, pitch and yaw, could be located within the base, or whencoupled with a user's wrist to provide a frame of reference, in whichcase one or more of the sensors for pitch, roll and yaw can beincorporated into the coupling. Examples of these arrangements are shownin later figures.

In the following description, the first control member may be generallyreferred to as a “joystick” or “control stick,” as it resemblesstructurally a portion of previously known types of joysticks, at leastwhere it is gripped, and functions, in some respects, as a might othertypes of joysticks because it is intended to be gripped by a person'shand and displaced (translated and/or rotated) or otherwise moved toindicate pitch, roll, and yaw, or motion. However, it should not implyany other structures that might be found in a conventional or otherjoysticks, and is intend only to signify an elongated structural elementthat can be gripped.

Referring now to the embodiment of FIGS. 7 and 8A, 8B, and 8C,controller 700 comprises a first control member, which may be referredto a joystick, having a pistol-grip-shaped body 702 formed by a gripportion 703, where it can be gripped at least two or more of the thumband third, fourth and fifth fingers of a hand, and a top portion 705located above where it is gripped. Within the first control member areone or more an integrated inertial measurement units (IMU) 704(indicated only schematically with dashed lines because the internalstructure with the body 702 is not visible in this view) to sense pitch,roll, and yaw control of the first control member. This embodimentincludes an optional quick-connection 718 for connecting to a base orother structure. This particular embodiment also incorporates optionalbuttons, such as trigger 706 (positioned for operation by an indexfinger) and attitude hold button 708. That can be operated by digits onthe hand holding the controller or by the user's other hand.

Mounted on top of the first control member, in a position that can bemanipulated by a thumb of a person gripping the body 702 of the firstcontrol member is mounted is a second control member. The second controlmember comprises a gantry arrangement 710 for the user to displace foreand aft, and left to right, to generate an input to indicate movementalong a y-axis and an x-axis, as well displace up or down to generate aninput to indicate movement along a z-axis. In this particular example,the gantry arrangement 710 is mounted on a platform 712 that moves thegantry arrangement up and down. Although different ways of moving theplatform (or the gantry 710), up and down can be employed, thisparticular example places the gantry 710 at one end of the hingedplatform 712. This allows the gantry arrangement to move up and withrespect to the first control member. Pushing down on the gantrydisplaces the platform 712 downwardly, thereby indicating an input forZ-axis control, while pulling up on the thumb loop (not shown) moves inthe opposite direction along the Z-axis.

Part of the Z-axis input arrangement on this controller also includes inthis example a third control member 714. In this example the thirdcontrol member takes the form of a paddle 716 where the third, fourthand/or fifth finger on a user's hand is located when gripping the firstcontrol member around the body 702, so that the paddle 716 can beselectively squeezed by the user when gripping the controller. Thepaddle 716 and the platform 712 can be spring loaded so that they are ina zero position to allow for z-axis input to indicate motion in eitherdirection from the zero position. The third control member acts as asecondary Z-axis control. The third control member is linked or coupledwith the second control member. The inclusion of a third control member,such as the finger paddle 716, “balances” the second control member,helping to relieve hitchhiker thumb fatigue in the user and gives finermotor control of user input along the Z-axis (up/down) while allowingalso for simultaneous movement of the gantry along the X-axis andY-axis.

FIGS. 8A and 8B show controller 700 with a number of elements removed tomore clearly show the cooperative movement of the paddle 716 andplatform 712. In FIG. 8A, the platform is in a fully depressed position,and in FIG. 8B the platform 712 is in a fully extended position, thedifference corresponding to the full travel of the second control memberalong a z-axis. In FIG. 8A the paddle 716 is in a fully extendedposition with respect to the body 702, and in FIG. 8B is fully depressedwith respect to the body at 702.

As shown in FIG. 8C, which is a perspective view of the controller 700with one-half of the body removed along with most of its other internalcomponents to reveal one example of a mechanical linkage. In thisexample, paddle 716 pivots about a pivot axis 720. A lever 722 connectedwith the paddle 716, but opposite of it with respect to the pivot axis720, is pivotally connected to a linkage 724. The other end of linkage724 is connected to a lever arm 726, to which platform 712 is connected.Platform 712 pivots about a pin forming an axis 728. Although not shownin the figure, a spring can be placed in an area indicated by referencenumber 730 to bias the paddle 716, and thus the entire linkage, toward azero or neutral position. Additional springs can also be used to providebalance and to bias the linkage to place the paddle and gantry in thezero positions on the Z-axis.

Turning to FIGS. 9, 10 and 11, controllers 900, 1000, and 1100 share thesame external components that make up the first and third controlmembers. Each has body 902 that forms the first control member and has,generally speaking, a shape like a joystick or pistol-grip that isintended to be gripped and held in the hand of a user. Eachincorporates, like controller 700, paddles 904 (which pivot from thetop, for example) that can be operated by one or more of the fingers ofthe user that is gripping the first control member. Each also has aprogrammable button 905, for which a second finger loop can besubstituted.

Similarly, each has a second control member on top of the body. Eachsecond control member includes a platform 906 that moves up and down (byway of a hinge or other mechanism) to provide the Z-axis input. However,each differs in the nature of the second control member. Controller 900uses a thumb loop 908 mounted to a gantry (TBD #) to provide x and yaxis input, while also enabling force to be applied to the z-axisplatform 906. This thumb loop can, preferably, be made in differentsizes using an insert (not shown) that can accommodate different sizes.(The thumb loops shown on other controllers in this disclosure can alsobe made resizable using an insert, if desired.) Controller 1000 of FIG.10 uses a gantry-type control member 1002, similar to the one shown onFIG. 7. And controller 1100 of FIG. 11 uses a trackball 1102 mounted onplatform 906 for x and y axis input. Pushing down on the track ball is az-axis input. The paddle 904 is used to provide input in the otherdirection along the z-axis.

In each of the controllers 900, 1000, 1100, as well as the handcontrollers illustrated in the remaining figures, the second and thirdcontrol members are coupled by a mechanical linkage disposed within thebody of the first control member, like linkage shown in FIG. 8C. Thelinkage of FIG. 8C is, however, intended to be representative of suchlinkages in general, as different arrangements and numbers of links canbe used depending on the particular geometries of the various parts andelements. Although other types of couplings or transmissions could beused to transmit displacement and force between the primary andsecondary z-axis control elements in any of the controllers shown anddescribed in FIGS. 7-20B. These could be other types of types ofmechanical transmissions (for example cables), as well as electrical andmagnetic transmissions that transmit position and, optionally, force,and combination any two or more of these types. A mechanical linkage,however, has an advantage since it is relatively simple and reliable forproviding a direct coupling between the two control members, and sinceit immediately communicates force and position to provide a comfortabledynamic balance.

Furthermore, all of the controllers shown in FIGS. 7-11, as well asthose shown in FIGS. 12-21, preferably have re-centering mechanisms foreach degree of freedom to give the user a sense of “zero” or nullcommand. When a control member is displaced along one of the degrees offreedom, it preferably generates a tactile feedback, such as force,shake or other haptic signal, of the control members to return them to aposition for zero input (the zero position). The mechanisms can consistof a spring that simply reacts with a spring force, or they can beactive systems that sense displacement and/or force, and generate areactive motion, force, other type of vibration haptic feedback, orcombination of them.

Although not shown in FIGS. 7-11, each of the controllers 700, 900, 1000and 1100, as well as the other controllers shown in the remainingfigures, include at least the elements shown in FIG. 1. For example, itincludes sensors (for example, inertial measurement units,potentiometers, optical encoders, or the like) for sensing displacementof the first, second and third control members; a processor forprocessing signals from the sensors; and a transmitter for transmittingthe input signals from the controller, which can be radio frequency,optical or wired (electrical or optical). Such sensors can take the formof inertial measurement units, potentiometers, optical encoders and thelike.

In any of the embodiments of controllers described in connection withFIGS. 1 to 21, user feedback can be supplied from the controller by oneor more of a number of mechanisms. For example, haptic vibration canprovide a subtle vibration feedback. Force feedback can provide feedbackin some or all degrees of freedom. Ambient heat and air can provideradiant heating and blowing air. Virtual reality multi-sensoryintegration can generate precise control within the virtual world.Integrated audio can provide sound feedback from a control target, suchas a drone or other target device. The controller can also providesurface heat and cold to give feedback through a heat and coolingsensation. The user interface (UI/UX) can include an integratedtouchscreen and visual indicators such as light, flashing colors, and soon.

Turning now to FIGS. 12, 13, and 14, shown are three variations of basestructures 1200, 1300 and 1400 to which any one of controllers 700, 900,1000, and 1100 can be connected. Those shown in any of the otherfigures, could be adapted as well. In the figures, controller 900 isused as an example, but the other controllers could be adapted for usewith any of the bases.

FIG. 12 shows a mobile, two-handed controller system. A two-handedcontroller provides a consistent, known reference frame (stabilized bythe non-dominant hand) even while moving, e.g., walking, skiing,running, driving. For certain types of applications, for exampleinspection, security and cinematographic drone missions, a handcontroller may be mounted on a platform that can be held or otherwisestabilized by the user's other hand. The platform may include secondarycontrols and, if desired, a display unit. The platform is stabilized bythe non-dominant hand. In one example, all 6-DoF inputs can be reactedthrough the platform. With such an arrangement, this example of acontrol system facilitates movement through the air like a fighter pilotwith intuitive (non-deliberate cognitive) inputs.

A hand controller, such as hand controller 900, is plugged (oralternatively, permanently mounted), into the top surface of the base. Ahandle or grip 1204 in the shape of, for example, a pistol grip, isprovided on the opposite side of the base for the user's other hand togrip while using the hand controller 900. (Other shapes and types ofhandles can also be envisioned by anyone skilled in the art.) Thisallows the user's other hand most likely the non-dominant hand, tostabilize the base. The base may, optionally, incorporate additionaluser interface elements 1206 and 1208, such as keys, buttons, dials,touchpads, trackpads, trackballs balls, etc. Display 1210 is mounted on,or incorporated into, the base in a position where the user can view it.One or more videos or graphical images from the application beingcontrolled can be displayed in real time on the display, such as livevideo from a drone, or a game. Alternatively, the base may include amount on which a smartphone or similar device can be placed or mounted.Alternate or optional features include one or a combination of any twoor more of the following features. The base can be reconfigurable foreither hand with a quick disconnect for the joystick and two mountingpoints. It can be either asymmetric (as shown) or symmetric in shape,with ample room for secondary controls. It can include a smartphoneattachment with tilt capability on its top surface. It may includesecondary joystick to allow for pan and tilt control of the dronecamera, and a capacitive deadman switch (or pressure deadman switch). Itmay also include large display mount and surface area for secondarycontrols. In an alternative embodiment a grip or handle can be locatedmore midline to the controller, thus reducing some off-axis moments.

FIG. 13 is an example of a base that can be moved to provide anotherinput, in this case it is a mouse 1300 with additional input buttons1304 and 1306. In this example, a secondary connection point 1308 for ahand controller is provided to accommodate both left and right-handedusers. One example would be for navigation through 3-D images on acomputer screen, where traditional mouse features would be used to movea cursor in the field of view, and to manipulate drop-down menus, whilethe controller 900 would be used to reorient and/or move the 3-D objectin multiple degrees of freedom of motion.

FIG. 14 shows an example of a wired, fixed base, single handedcontroller 1400.

Although not required, each of the figures show an example embodiment inwhich the controller can be quickly connected at its bottom to the base.In each example of a base, the controller is connected to ajoystick-like, small lever (1202, 1302 and 1402). This lever could beused to provide pitch, roll and yaw input, with sensors located withinthe base, but it does not have to be. It can instead (or in addition) beused to center the first control member at a zero position and providefeedback to the user. An RF or wired connection between the controllerand the base can be used to communicate signals from sensors within thecontroller.

FIG. 15 shows an example of an embodiment of a hand controller 1500,like controller 900, that includes an index finger loop 1502 in additionto a thumb loop 1503 that functions as a second control member. Thisindex finger loop can be used to control opening and closing a physicalor virtual end effector, say a hand grasp on an object in a virtualworld. The design can ergonomically fit within the palm of the hand invery low profile and can be optimized for, virtual/augmented reality ordrone flight. The addition of an index finger loop to open and close anend effector, for example, can benefit virtual/augmented realityapplications.

Also schematically shown in FIG. 15 is an attachment 1504 for placementon a forearm 1506 of a user. A coupling 1508 between the attachment 1504and the hand controller 1500 supports the hand controller and allows foruse of potentiometers or optical encoders to precisely measure angulardisplacement of pitch, roll, and yaw of controller 1500 when it isconnected to a pivot point 1510 that is in a fixed relation to theforearm attachment 1504, even if removed from a base station. Theindexing off of the wrist or forearm allows for this. In one embodiment,the hand controller does not use an IMU to sense one or more of thepitch, roll or raw, using instead the other types of sensors.Alternately the system can use two or more IMUs and software filteringof the data to measure relative displacement and to command flightcontrol.

Moving any point of reference through physical or virtual space by wayof a hand controller requires constant insight into displacement inevery degree of freedom being controlled. Stated differently, it isimportant to know where “zero input” is at all times for movement alongx, y, and z directions and yaw for a drone. Other flight regimes, suchas virtual and augmented reality, computer gaming and surgical roboticsmay require as many as six independent degrees of freedom simultaneously(X, Y, Z, pitch, yaw, roll). Moreover, for drone flight and virtualreality and augmented reality in particular, the ability to be mobilewhile maintaining precise control of the point of reference (POR) isdesirable.

FIGS. 15 to 20B illustrate several, representative embodiments ofcontrol systems having two parts: a hand-held controller and a forearmattachment in the form of a brace adapted or configured for mounting toa forearm or wrist of the user that provides a consistent, knownreference frame (anchored to a user's wrist) even while the user or theuser's arm is moving or accelerating, such as by walking, skiing,running, or driving.

In the examples shown in these figures, the forearm attachment mighttake any one of a number of forms. For example, it might comprise abrace, wrist wrap (which can be wrapped around a forearm or wrist andfastened using, for example, Velcro), slap-bracelet, or other items thatconforms to at least a portion of the forearm. However, it may alsocomprise a relatively stiff support structure. The forearm attachmentmay be referred to as a brace, cuff or “gauntlet” because, structurallyand/or functionally, it resembles these items in some respects. However,use of these terms should not imply structures beyond what is shown orrequired for the statement function.

The hand controller and the forearm attachment are connected by amechanical linkage, strut or support. In one embodiment, it is a passivelinkage, that it other embodiments it is not. One type of passivemechanical linkage used in the examples described below is a two-axisgimbal pivot with centering springs and potentiometers to measuredisplacement. Alternately, cables, double piston mechanisms (compressionsprings), pneumatic cylinders or passive stiffeners/battens, possiblybuilt into a partial glove, could be used. In the examples, the linkageimparts a force to the user with which the user can sense zero input atleast one, or at least two, or in all three axes of rotation on thejoystick.

Small inertial measurement units (IMUs) may also be placed within thecontroller and mounted to the wrist or forearm using the forearmattachment, for example, allowing detection of pure differential(relative) motion between the forearm and the controller. Noisy signalswould be managed by oversampling and subsequent decimation withdigital+/−adaptive filtering, thereby achieving measurement of relativemotion of the hand versus the arm in mechanically noisy environments(while hiking, running or otherwise moving). However, the embodimentsthat can measure one or more of pitch, roll or yaw may be able to limitthe use of an IMU to sensing just one or two rotational inputs, or omitthem altogether from the controller.

In an alternative embodiment, an active mechanical feedback can be usedto inform the user of displacement in a given axis of rotation mightalso include vibration haptics and force feedback.

For drone flight, one embodiment involves two gimbaled degrees offreedom at the wrist, and two at the thumb: wrist pitch (X orforwards/backwards) and wrist yaw (pivot left/right); thumb/Z paddle(translate up/down) and thumb Y (translate left/right).

Alternately, it is possible to record displacement in roll of theforearm as well, but it requires a gauntlet that extends at least halfway up the forearm and perhaps more. A full 6-degrees of freedomcontrol, including measurement forearm roll, isn't necessary for droneflight, although it would be desirable for augmented realityapplications. The yaw and Y translation inputs described above might beswapped, at user preference, based on flight testing and personalpreference.

The thumb loop/“Z paddle” is preserved while using a “gantry” on top ofthe joystick to measure intended displacement laterally. Other methodsof measuring forearm roll might include EMG detection of forearm muscleelectrical potential, a conformal forearm wrap with pressure sensorsthat pick up differential contours of the forearm as a function ofrotation, and differential IMUs or a combination of an IMU and a camerasystem (wrist vs elbow), showing rotation. The latter solutions wouldlikely require vibration haptics or force feedback to inform the user ofthe zero position in roll.

One or more of the following features may be incorporated:reconfigurable for either hand; symmetric shape with buttons availablefrom either side; quick don and doff of wrist wrap or disconnect ofjoystick; smartphone attachment with tilt capability on wrist wrap;secondary joystick at the base of the joystick to allow for (pan)/tiltof the drone camera; secondary joystick retracts/extends from base ofjoystick like a ball point pen; capacitive Deadman Switch (or PressureDeadman Switch)—fail safe; a modular joystick that is able to remove andplace on tabletop base, operate standalone or on other types offunction-specific bases, such as those described above.

Gimbal pivots shown in the drawings contain centering torsion springsand potentiometers. Preferably, couplings or linkages that connect thejoystick to that the gimbals are designed to be to be adjustable fordifferent sized users.

A universal smart phone holder may also attached to a bracket mounted tothe forearm attachment or brace.

The hand controllers in the following figures comprise sixdegrees-of-freedom single hand control device, with first control memberin the form of joystick (or joystick like device), and second controlmember for the user's thumb (whether a loop, gantry, track ball, touchpad or other input device) has its Z-axis travel augmented by otherthird control member configured to be used by one or more non-indexfingers of the same hand and that move in conjunction with, and inopposition to, the second control member.

Further features useful in, for example, applications to drone flight orto virtual/augmented reality, can include a forearm brace to allowmobile potentiometer or optical encoder sensing of pitch, roll, and yaw;pan/tilt controls can be integrated into the controller, as can a smartdevice (smartphone, tablet) holder. A base structure to which the handcontroller is attached can also include a second handle (for thenon-dominant hand) to allow for mobile potentiometer or optical encodersensing.

Alternate solutions for yaw precision can include one or more of:induced magnetic field wrist bracelet, differential IMUs, softwarefiltering of the IMU to reduce yaw related noise, reaction wheels (highprecision gyro), and inertial (high precision yaw gyro) balanced yawwith potentiometers or optical encoders. Software filtering of IMU datacan include dynamic re-zeroing.

The control signals from the controller can be further augmented byadditional inputs. For example, a head or body mounted “connect sensor”can be used. This could use a grid-type infrared input or otheroptically based variations, such as RF directional or omnidirectionaltracking. The connect sensors could be head mounted, such as forinteractive virtual reality applications, or wrist mounted. Magnetic“dot” tracking can be used for more general body position inputs.

Referring now to FIG. 16, controller 1600 is substantially similar toother hand controllers described in the preceding paragraphs. In thisexample, it is connected to a forearm attachment 1602 that includes avideo display 1604 and additional user inputs 1606 in the form ofbuttons and other types of user input. Connection 1608 between thecontroller 1600 and the forearm attachment 1602 is a relatively stifflinkage that maintains the relative position of controller 1600 with theforearm attachment 1602, provide a pivot point around which pitch, yaw,and roll can be measured using either internal sensors or externalsensors mounted at the end of connection 1608.

Referring now to FIGS. 17 and 18, which illustrate an alternateembodiment of a cuff 1700 that acts as a forearm attachment. In thisexample, hand controller 1702 is schematically represented. It isrepresentative of any of the hand controllers that have been describedherein. Any of the hand controllers described herein can be adapted foruse in this example. In this example, the controller is connected with apitch sensor 1706 that is located below the controller and attached tothe cuff 1700 with a mechanical link or strut 1708 that it is adjustableas indicated by length adjustment 1710. The end of the mechanical link1708 is attached to the forearm attachment using a spherical bearing1712 to allow for different angles. Like the length adjustment 1710, itwill be tightened down once the user adjust the position of thecontroller to their satisfaction.

This example contemplates that an IMU is not used in the controller, atleast for pitch and yah measurements. Rather, a yaw, roll and pitchsensors are incorporated into the bottom of the hand controller 1702, orthe base 1703 of a mechanical connection or support between the forearmattachment and the controller. Such sensors can take, in one example,the form of gimbal with a potentiometer and a torsion spring to providefeedback from zero position. In this example, pitch sensor isincorporated into the bottom of the controller 1702, though it couldalso be incorporated into the base of the link or strut 1708 in whichthe pitch sensor 1706 is placed. A roll sensor, which is not visible,can be placed in either the base of the linkage 1708, in which the pitchsensor is placed, or in the bottom or base portion 1704 of thecontroller 1702.

Referring now to FIGS. 19A, 19B, 19C and 19D, illustrated is anembodiment of a control system 1900 with a specific example of a doublegimbal link 1902 between a forearm attachment 1904 and a hand controller1906 (FIG. 19D only.). The double gimbal link 1902 attaches gimbals 1908and 1910 placed at ninety degrees to each other to measure,respectively, pitch and yaw. The hand controller is connected to handcontroller mount 1912 which acts as a lever arm and is connected to yawgimbal 1910. The forearm attachment, which includes a sleeve or brace1914, to which a strap may be connected to attach it to the arm, issupported on a lever arm 1916 that is connected to one side of the pitchgimbal 1908. Note that, in FIG. 19C. the hand controller mount 1912 thatis shown is a variation of the one shown in FIGS. 19A and 19B, in thatit is adjustable. A phone holder 1918 may be mounted or attached to thearm attachment 1904 so that it can be seen by the user. The phone holderis adjustable in this example so that hold different types and sizes ofphones.

Turning now to FIGS. 20A and 20B, shown is another example of a controlsystem similar to the one of FIGS. 19A-19D. In this example, the controlsystem uses a pitch gimbal 2002 and a yaw gimbal 2004, which measurepitch and yaw, respectively, connected with a bracket 2006, in a mannersimilar to that shown in FIGS. 19A-19D. The pitch gimbal 2002 is mountedto a forearm attachment in the form of a brace 2008 placed near wherethe wrist joint pivots when gripping and rotating the controller 2010.The brace is held on by a strap 2012. The brace, as in the forgoingembodiments, acts as a stabilizer. The controller 2010 is mounted to anadjustable length lever arm 2014. In this example, controller 2010, likeother hand controllers in the foregoing embodiments, has a body 2016that forms a first control member that is graspable by the user that isused to input rotational displacements (two of which are measure by thegimbals), a second control member on top of the body 2016 in the form ofa thumb loop 2018 for X, Y, Z input. On the front, near the bottom, ofthe body is a joy stick 2022, which can be used as input for camera panand tilt, for example, or to manipulate tools.

Thus, systems and methods have been described that that include acontroller that allows a user to provide rotational and translationalcommands in six independent degrees of freedom using a single hand. Thesystem and method may be utilized in a wide variety of controlscenarios. While a number of control scenarios are discussed below,those examples are not meant to be limiting, and one of ordinary skillin the art will recognize that many control scenarios may benefit frombeing able to provide rotational and translational movement using asingle hand, even if fewer than all control outputs for all six degreesof freedom are required.

In an embodiment, the control systems and methods discussed above may beutilized in a wide variety of medical applications. While a number ofmedical applications are discussed below, those examples are not meantto be limiting, and one of ordinary skill in the art will recognize thatmany other medical applications may benefit from being able to providerotational and translational movement using a single hand. Furthermore,in such embodiments, in addition to the rotational and translationalmovement provided using first and second control members discussedabove, control buttons may be configured for tasks such as, for example,end-effector capture, biopsy, suturing, radiography, photography, and/ora variety of other medical tasks as may be known by one or more ofordinary skill in the art.

For example, the control systems and methods discussed above may providea control system for performing laparoscopic surgery and/or a method forperforming laparoscopic surgery. Conventional laparoscopic surgery isperformed using control systems that require both hands of a surgeon tooperate the control system. Using the control systems and/or the methodsdiscussed above provide several benefits in performing laparoscopicsurgery, including fine dexterous manipulation of one or more surgicalinstruments, potentially without a straight and rigid path to the endeffector.

In another example, the control systems and methods discussed above mayprovide a control system for performing minimally invasive or naturalorifice surgery and/or a method for performing minimally-invasive ornatural-orifice surgery. Conventional minimally invasive or naturalorifice surgery is performed using control systems that require bothhands of a surgeon to operate the control system. Using the controlsystems and/or the methods discussed above provide several benefits inperforming minimally invasive or natural orifice surgery, including finedexterous manipulation of one or more surgical tools, potentiallywithout a straight and rigid path to the end effector.

In another example, the control systems and methods discussed above mayprovide a control system for performing prenatal intrauterine surgeryand/or a method for performing prenatal surgery. Conventional prenatalsurgery is performed using control systems that require both hands of asurgeon to operate the control system in very tight confines. Using thecontrol systems and/or the methods discussed above provide severalbenefits in performing prenatal surgery, including fine dexterousmanipulation of one or more surgical tools, potentially without astraight and rigid path to the end effector.

For any of the above surgical examples, the control systems and methodsdiscussed above may provide a very stable control system for performingmicroscopic surgery and/or a method for performing microscopic surgery.Using the control systems and/or the methods discussed above provideseveral benefits in performing microscopic surgery, including highlyaccurate camera and end effector pointing.

In another example, the control systems and methods discussed above mayprovide a control system for performing interventional radiology and/ora method for performing interventional radiology. Conventionalinterventional radiology is performed using control systems that requireboth hands of a surgeon to operate the control system. Using the controlsystems and/or the methods discussed above provide several benefits inperforming interventional radiology, including highly accuratenavigation through for interventional radiology. In another example, thecontrol systems and methods discussed above may provide a control systemfor performing interventional cardiology and/or a method for performinginterventional cardiology. Conventional interventional cardiology isperformed using control systems that require both hands of aninterventionist to operate the control system. Using the control systemsand/or the methods discussed above provide several benefits inperforming interventional cardiology, including highly accuratenavigation through the vascular tree using one hand.

In another example, the control systems and methods discussed above mayprovide a control system including Hansen/Da Vinci robotic controland/or a method for performing Hansen/Da Vinci robotic control.Conventional Hansen/Da Vinci robotic control is performed using controlsystems that require both hands of a surgeon to operate the controlsystem. Using the control systems and/or the methods discussed aboveprovide several benefits in performing Hansen/Da Vinci robotic control,including fluid, continuous translation and reorientation withoutshuffling the end effector for longer motions.

In another example, the control systems and methods discussed above mayprovide a control system for performing 3D- or 4D-image guidance and/ora method for performing 3D- or 4D-image guidance. Conventional 3D- or4D-image guidance is performed using control systems that require bothhands of a surgeon to operate the control system. Using the controlsystems and/or the methods discussed above provide several benefits inperforming 3D- or 4D-image guidance, including fluid, continuoustranslation and reorientation without shuffling the end effector forlonger motions.

In another example, the control systems and methods discussed above mayprovide a control system for performing endoscopy and/or a method forperforming endoscopy. Conventional endoscopy is performed using controlsystems that require both hands of a surgeon to operate the controlsystem. Using the control systems and/or the methods discussed aboveprovide several benefits in performing endoscopy, including fluid,continuous translation and reorientation without shuffling the endeffector for longer motions. This also applies to colonoscopy,cystoscopy, bronchoscopy, and other flexible inspection scopes.

In an embodiment, the control systems and methods discussed above may beutilized in a wide variety of defense or military applications. While anumber of defense or military applications are discussed below, thoseexamples are not meant to be limiting, and one of ordinary skill in theart will recognize that many other defense or military applications maybenefit from being able to provide rotational and translational movementusing a single hand.

For example, the control systems and methods discussed above may providea control system for unmanned aerial systems and/or a method forcontrolling unmanned aerial systems. Conventional unmanned aerialsystems are controlled using control systems that require both hands ofan operator to operate the control system. Using the control systemsand/or the methods discussed above provide several benefits incontrolling unmanned aerial systems, including intuitive single-handed,precise, non-cross-coupled motion within the airspace.

In another example, the control systems and methods discussed above mayprovide a control system for unmanned submersible systems and/or amethod for controlling unmanned submersible systems. Conventionalunmanned submersible systems are controlled using control systems thatrequire both hands of an operator to operate the control system. Usingthe control systems and/or the methods discussed above provide severalbenefits in controlling unmanned submersible systems, includingintuitive single-handed, precise, non-cross-coupled motion within thesubmersible space. As shown in FIG. 20B, the control system can includea smart device (e.g., smartphone, tablet) holder 2024.

In another example, the control systems and methods discussed above mayprovide a control system for weapons targeting systems and/or a methodfor controlling weapons targeting systems. Conventional weaponstargeting systems are controlled using control systems that require bothhands of an operator to operate the control system. Using the controlsystems and/or the methods discussed above provide several benefits incontrolling weapons targeting systems, including precise, intuitive,single-handed targeting.

In another example, the control systems and methods discussed above mayprovide a control system for counter-improvised-explosive-device (IED)systems and/or a method for controlling counter-IED systems.Conventional counter-IED systems are controlled using control systemsthat require both hands of an operator to operate the control system.Using the control systems and/or the methods discussed above provideseveral benefits in controlling counter-IED systems, including precise,intuitive, single-handed pointing or targeting.

In another example, the control systems and methods discussed above mayprovide a control system for heavy mechanized vehicles and/or a methodfor controlling heavy mechanized vehicles. Conventional heavy mechanizedvehicles are controlled using control systems that require both hands ofan operator to operate the control system. Using the control systemsand/or the methods discussed above provide several benefits incontrolling heavy mechanized vehicles, including precise, intuitive,single-handed targeting.

In another example, the control systems and methods discussed above mayprovide a control system for piloted aircraft (e.g., rotary wingaircraft) and/or a method for controlling piloted aircraft. Conventionalpiloted aircraft are controlled using control systems that require bothhands of an operator to operate the control system. Using the controlsystems and/or the methods discussed above provide several benefits incontrolling piloted aircraft, including precise, intuitive,single-handed, non-cross-coupled motion within the airspace for thepiloted aircraft.

In another example, the control systems and methods discussed above mayprovide a control system for spacecraft rendezvous and docking and/or amethod for controlling spacecraft rendezvous and docking. Conventionalspacecraft rendezvous and docking is controlled using control systemsthat require both hands of an operator to operate the control system.Using the control systems and/or the methods discussed above provideseveral benefits in controlling spacecraft rendezvous and docking,including precise, intuitive, single-handed, non-cross-coupled motionwithin the space for rendezvous and/or docking.

In another example, the control systems and methods discussed above mayprovide a control system for air-to-air refueling (e.g., boom control)and/or a method for controlling air-to-air refueling. Conventionalair-to-air refueling is controlled using control systems that requireboth hands of an operator to operate the control system. Using thecontrol systems and/or the methods discussed above provide severalbenefits in controlling air-to-air refueling, including precise,intuitive, single-handed, non-cross-coupled motion within the airspacefor refueling.

In another example, the control systems and methods discussed above mayprovide a control system for navigation in virtual environments (e.g.,operational and simulated warfare) and/or a method for controllingnavigation in virtual environments. Conventional navigation in virtualenvironments is controlled using control systems that require both handsof an operator to operate the control system. Using the control systemsand/or the methods discussed above provide several benefits incontrolling navigation in virtual environments, including precise,intuitive, single-handed, non-cross-coupled motion within the virtualenvironment.

In an embodiment, the control systems and methods discussed above may beutilized in a wide variety of industrial applications. While a number ofindustrial applications are discussed below, those examples are notmeant to be limiting, and one of ordinary skill in the art willrecognize that many other industrial applications may benefit from beingable to provide rotational and translational movement using a singlehand.

For example, the control systems and methods discussed above may providea control system for oil exploration systems (e.g., drills, 3Dvisualization tools, etc.) and/or a method for controlling oilexploration systems. Conventional oil exploration systems are controlledusing control systems that require both hands of an operator to operatethe control system. Using the control systems and/or the methodsdiscussed above provide several benefits in controlling oil explorationsystems, including precise, intuitive, single-handed, non-cross-coupledmotion within the formation.

In another example, the control systems and methods discussed above mayprovide a control system for overhead cranes and/or a method forcontrolling overhead cranes. Conventional overhead cranes are controlledusing control systems that require both hands of an operator to operatethe control system. Using the control systems and/or the methodsdiscussed above provide a benefit in controlling overhead cranes wheresingle axis motion is often limited, by speeding up the process andincreasing accuracy.

In another example, the control systems and methods discussed above mayprovide a control system for cherry pickers or other mobile industriallifts and/or a method for controlling cherry pickers or other mobileindustrial lifts. Conventional cherry pickers or other mobile industriallifts are often controlled using control systems that require both handsof an operator to operate the control system, and often allowtranslation (i.e., x, y, and/or z motion) in only one direction at atime. Using the control systems and/or the methods discussed aboveprovide several benefits in controlling cherry pickers or other mobileindustrial lifts, including simultaneous multi-axis motion via asingle-handed controller.

In another example, the control systems and methods discussed above mayprovide a control system for firefighting systems (e.g., water cannons,ladder trucks, etc.) and/or a method for controlling firefightingsystems. Conventional firefighting systems are often controlled usingcontrol systems that require both hands of an operator to operate thecontrol system, and typically do not allow multi-axis reorientation andtranslation. Using the control systems and/or the methods discussedabove provide several benefits in controlling firefighting systems,including simultaneous multi-axis motion via a single-handed controller.

In another example, the control systems and methods discussed above mayprovide a control system for nuclear material handling (e.g.,gloveboxes, fuel rods in cores, etc.) and/or a method for controllingnuclear material handling. Conventional nuclear material handlingsystems are controlled using control systems that require both hands ofan operator to operate the control system. Using the control systemsand/or the methods discussed above provide several benefits incontrolling nuclear material handling, including very precise, fluid,single-handed, multi-axis operations with sensitive materials.

In another example, the control systems and methods discussed above mayprovide a control system for steel manufacturing and other hightemperature processes and/or a method for controlling steelmanufacturing and other high temperature processes. Conventional steelmanufacturing and other high temperature processes are controlled usingcontrol systems that require both hands of an operator to operate thecontrol system. Using the control systems and/or the methods discussedabove provide several benefits in controlling steel manufacturing andother high temperature processes, including very precise, fluid,single-handed, multi-axis operations with sensitive materials.

In another example, the control systems and methods discussed above mayprovide a control system for explosives handling (e.g., in miningapplications) and/or a method for controlling explosives handling.Conventional explosives handling is controlled using control systemsthat require both hands of an operator to operate the control system.Using the control systems and/or the methods discussed above provideseveral benefits in controlling explosives handling, including veryprecise, fluid, single-handed, multi-axis operations with sensitivematerials.

In another example, the control systems and methods discussed above mayprovide a control system for waste management systems and/or a methodfor controlling waste management systems. Conventional waste managementsystems are controlled using control systems that require both hands ofan operator to operate the control system. Using the control systemsand/or the methods discussed above provide several benefits incontrolling waste management systems, including very precise, fluid,single-handed, multi-axis operations with sensitive materials.

In an embodiment, the control systems and methods discussed above may beutilized in a wide variety of consumer applications. While a number ofconsumer applications are discussed below, those examples are not meantto be limiting, and one of ordinary skill in the art will recognize thatmany other consumer applications may benefit from being able to providerotational and translational movement using a single hand.

For example, the control systems and methods discussed above may providea control system for consumer electronics devices e.g., Nintendo Wii®(Nintendo of America Inc., Redmond, Wash., USA), Nintendo DS®, MicrosoftXBox® (Microsoft Corp., Redmond, Wash., USA), Sony Playstation® (SonyComputer Entertainment Inc., Corp., Tokyo, Japan), and other videoconsoles as may be known by one or more of ordinary skill in the art)and/or a method for controlling consumer electronics devices.Conventional consumer electronics devices are controlled using controlsystems that require both hands of an operator to operate the controlsystem (e.g., a hand controller and keyboard, two hands on onecontroller, a Wii® “nunchuck” z-handed I/O device, etc.) Using thecontrol systems and/or the methods discussed above provide severalbenefits in controlling consumer electronics devices, including theability to navigate with precision through virtual space with fluidity,precision and speed via an intuitive, single-handed controller.

In another example, the control systems and methods discussed above mayprovide a control system for computer navigation in 3D and/or a methodfor controlling computer navigation in 3D. Conventional computernavigation in 3D is controlled using control systems that either requireboth hands of an operator to operate the control system or do not allowfluid multi-axis motion through space. Using the control systems and/orthe methods discussed above provide several benefits in controllingcomputer navigation in 3D, including very precise, fluid, single-handed,multi-axis operations.

In another example, the control systems and methods discussed above mayprovide a control system for radio-controlled vehicles and/or a methodfor controlling radio-controlled vehicles. Conventional radio-controlledvehicles are controlled using control systems that require both hands ofan operator to operate the control system. Using the control systemsand/or the methods discussed above provide several benefits incontrolling radio-controlled vehicles, including intuitivesingle-handed, precise, non-cross-coupled motion within the airspace forradio-controlled vehicles.

In another example, the control systems and methods discussed above mayprovide a control system for 3D computer aided drafting (CAD) imagemanipulation and/or a method for controlling 3D CAD image manipulation.Conventional 3D CAD image manipulation is controlled using controlsystems that either require both hands of an operator to operate thecontrol system or do not allow fluid multi-axis motion through 3D space.Using the control systems and/or the methods discussed above provideseveral benefits in controlling 3D CAD image manipulation, includingintuitive single-handed, precise, non-cross-coupled motion within the 3Dspace.

In another example, the control systems and methods discussed above mayprovide a control system for general aviation and/or a method forcontrolling general aviation. Conventional general aviation iscontrolled using control systems that require both hands and feet of anoperator to operate the control system. Using the control systems and/orthe methods discussed above provide several benefits in controllinggeneral aviation, including intuitive single-handed, precise,non-cross-coupled motion within the airspace for general aviation.

It is understood that variations may be made in the above withoutdeparting from the scope of the invention. While specific embodimentshave been shown and described, modifications can be made by one skilledin the art without departing from the spirit or teaching of thisinvention. The embodiments as described are exemplary only and are notlimiting. Many variations and modifications are possible and are withinthe scope of the invention. Furthermore, one or more elements of theexemplary embodiments may be omitted, combined with, or substituted for,in whole or in part, with one or more elements of one or more of theother exemplary embodiments. Accordingly, the scope of protection is notlimited to the embodiments described, but is only limited by the claimsthat follow, the scope of which shall include all equivalents of thesubject matter of the claims.

What is claimed is:
 1. A controller, comprising: a first control membermovable with three degrees of freedom and providing in response theretoa first set of up to three independent control inputs, each indicatingan amount of displacement of the first control member in one of thethree degrees of freedom relative to a null position, wherein the firstcontrol member is configured to be gripped by a user's single hand, thefirst control member including a first inertial motion sensor; a secondcontrol member extending from the first control member that is movablewith at least one independent degree of freedom relative to butindependently of the first control member and providing in responsethereto a second set of at least one independent control inputsindicating an amount of displacement of the second control member ineach of the at least one independent degrees of freedom relative to thefirst control member, where each control inputs of the second set isindependent of each of the control input of the first set, wherein thesecond control member is configured to be manipulated by the a digit ofthe user's single hand; a forearm brace with which first control memberis coupled; and a second inertial motion sensor operatively connected tothe forearm brace, wherein the first inertial motion sensor and thesecond inertial motion sensor being configured to provide datarepresentative of relative movement between the first control member andthe brace, wherein at least one of the first set of three independentcontrol inputs is provided in response to the movement of the firstcontrol member in at least one of the three degrees of freedom relativeto the forearm brace.
 2. The controller of claim 1, further comprising alinkage extending from the forearm brace, to which the first controlmember is coupled in a fixed relationship.
 3. The controller of claim 2,wherein the linkage comprises at least two members that articulate abouta joint with an axis of rotation that, when the forearm brace is mountedon a user's forearm and the first control member is gripped by theuser's hand, extends through the user's wrist, wherein movement of thefirst control member about the axis of rotation corresponds to one ofthe three degrees of freedom of the first control member.
 4. Thecontroller of claim 3, further comprising a sensor for measuring angulardisplacement of the joint for generating a signal indicative of theamount of the displacement, wherein a first one of the first set ofcontrol inputs that indicates the amount of movement of the firstcontrol member in the first one of the three degrees of freedom respondsto the measured angular displacement.
 5. The controller of claim 1,wherein the first control member has an elongated shape with a portionadapted for being gripped by the user's hand.
 6. The controller of claim5, further comprising a third control member on the portion adapted forbeing gripped for being depressed by one or more digits of the user'shand while gripping the first control member, wherein the third controlmember is movable in conjunction with, and in opposition to, movement ofthe second control member in the at least one independent degree offreedom.
 7. A controller comprising: a first control member shaped to begrasped by a hand of a user, the controller being displaceable by theuser in three degrees of freedom of movement about each of three axes ofrotation, the controller having at least two control inputs, each ofwhich indicates an amount of angular displacement of the first controlmember about a separate one of the three axes of rotation relative to anull position, the first control member including a first inertialmotion sensor; a second control member mounted on the first controlmember for displacement relative to the first control member in at leastone degree of freedom of movement, the second control member beinglocated on the first control member in a position that allows formovement by a thumb or index finger on the user's hand while grippingthe first control member; and a frame configured for removably mountingon a part of a user's arm for serving as a reference for at least one ofthe three rotational axes, a second inertial motion sensor operativelyconnected to the frame, and configured to measure relative movement ofthe frame and the first control member in at least one of the threerotational degrees of freedom of movement of the controller to generateat least one of the at least two control inputs.
 8. The controller ofclaim 7, further comprising a linkage coupling the frame and the firstcontrol member for determining movement of the first control memberrelative to the user's forearm in at least one degree of freedom.
 9. Thecontroller of claim 8, wherein the linkage comprises at least twomembers that articulate about a joint with an axis of rotationcorresponding to one of the three axes of rotation of the first controlmember and that extends through the user's wrist when the frame ismounted on the user's forearm and the first control member is gripped bythe user's hand.
 10. The controller of claim 9, further comprising asensor for measuring angular displacement of the joint for generating asignal indicative of the amount of the displacement of the joint andgenerating in response thereto one of the at least two control inputs.11. The controller of claim 9, wherein the linkage further comprises athird member and a second joint having an axis of rotation correspondingto a second one of the three axes of rotation of the first controlmember, the second joint articulating when a user moves the firstcontrol member about the second one of the three axes of rotation. 12.The controller of claim 8, wherein the linkage is adjustable foraccommodating different users.
 13. The controller of claim 7, whereinthe frame comprises a brace configured for attachment to the user'swrist or forearm.
 14. The controller of claim 13, wherein the framecomprises a bracket for holding a visual display in view of the userwhen the frame is mounted on the user's arm and the user is manipulatingthe controller.
 15. The controller of claim 7, wherein the first controlmember has an elongated shape with a portion adapted for being grippedby a user's hand.
 16. The controller of claim 7, further comprising athird control member for being depressed by one or more digits of auser's single hand while gripping the first control member, wherein thethird control member is movable in conjunction with, and in oppositionto, movement of the second control member in the at least one degree offreedom of movement.